Image retrieval from scientific publications: Text and image content processing to separate multipanel figures

Images contained in scientifi publications are widely considered useful for educational and research purposes, and their accurate indexing is critical for efficient and effective retrieval. Such image retrieval is complicated by the fact that figure in the scientifi literature often combine multiple individual subfigure (panels). Multipanel figure are in fact the predominant pattern in certain types of scientifi publications. The goal of this work is to automatically segment multipanel figures— necessary step for automatic semantic indexing and in the development of image retrieval systems targeting the scientifi literature. We have developed a method that uses the image content as well as the associated figur caption to: (1) automatically detect panel boundaries; (2) detect panel labels in the images and convert them to text; and (3) detect the labels and textual descriptions of each panel within the captions. Our approach combines the output of imagecontent and text-based processing steps to split the multipanel figure into individual subfigure and assign to each subfigur its corresponding section of the caption. The developed system achieved precision of 81% and recall of 73% on the task of automatic segmentation of multipanel figures